Introduction
This blog started off life as a blog post on how I use nix but somehow transformed itself into a “how I do data visualisation” blog post. The nix is still here though quietly doing its work in the background.
Suppose you want to analyze your local election results and visualize them using a choropleth but you’d like to use Haskell. You could try using the shapefile package but you will have to do a lot of the heavy lifting yourself (see here for an extended example). But you know folks using R can produce such things in a few lines of code and you also know from experience that unless you are very disciplined large R scripts can quickly become unmaintainable. Can you get the best of both worlds? It seems you can if you use inline-r. But now you have a bit of configuration problem: how do you make sure that all the Haskell packages that you need and all the R packages that you need are available and how can you make sure that your results are reproducible? My answer to use nix.
Acknowledgements
I’d like to thank all my colleagues at Tweag who have had the patience to indulge my questions especially zimbatm and also all folks on the #nixos irc channel.
Downloading and Building
The sources are on github.
You’ll have to build my Include pandoc filter somehow and then
pandoc -s HowIUseNix.lhs --filter=./Include -t markdown+lhs > HowIUseNixExpanded.lhs
BlogLiterately HowIUseNixExpanded.lhs > HowIUseNixExpanded.htmlAnalysing the Data
Before looking at the .nix file I use, let’s actually analyse the data. First some preamble
> {-# OPTIONS_GHC -Wall #-}
> {-# OPTIONS_GHC -fno-warn-unused-do-bind #-}
> {-# LANGUAGE TemplateHaskell #-}
> {-# LANGUAGE QuasiQuotes #-}
> {-# LANGUAGE DataKinds #-}
> {-# LANGUAGE FlexibleContexts #-}
> {-# LANGUAGE TypeOperators #-}
> {-# LANGUAGE OverloadedStrings #-}
> import Prelude as P
> import qualified Language.R as R
> import Language.R.QQ
> import System.Directory
> import System.FilePath
> import Control.Monad
> import Data.List (stripPrefix, nub)
> import qualified Data.Foldable as Foldable
> import Frames hiding ( (:&) )
> import Control.Lens
> import qualified Data.FuzzySet as Fuzzy
> import Data.Text (pack, unpack)
> import qualified Control.Foldl as Foldl
> import Pipes.Prelude (fold)
We can scrape the data for the Kington on Thames local elections and put it in a CSV file. Using the Haskell equivalent of data frames / pandas, Frames, we can “read” in the data. I say “read” in quotes because if this were a big data set (it’s not) then it might not fit in memory and we’d have to process it as a stream. I’ve included the sheet in the github repo so you can try reproducing my results.
> tableTypes "ElectionResults" "data/Election Results 2018 v1.csv"
> loadElectionResults :: IO (Frame ElectionResults)
> loadElectionResults = inCoreAoS (readTable "data/Election Results 2018 v1.csv")
We could analyse the data set in order to find the ward (aka electoral district) names but I happen to know them.
> wardNames :: [Text]
> wardNames = [ "Berrylands"
> , "Norbiton"
> , "Old Malden"
> , "Coombe Hill"
> , "Beverley"
> , "Chessington North & Hook"
> , "Tolworth and Hook Rise"
> , "Coombe Vale"
> , "St James"
> , "Tudor"
> , "Alexandra"
> , "Canbury"
> , "Surbiton Hill"
> , "Chessington South"
> , "Grove"
> , "St Marks"
> ]
We need a query to get both the total number of votes cast in the ward and the number of votes cast for what is now the party which controls the borough. We can create two filters to run in parallel, summing as we go.
> getPartyWard :: Text -> Text -> Foldl.Fold ElectionResults (Int, Int)
> getPartyWard p w =
> (,) <$> (Foldl.handles (filtered (\x -> x ^. ward == w)) .
> Foldl.handles votes) Foldl.sum
> <*> (Foldl.handles (filtered (\x -> x ^. ward == w &&
> x ^. party == p)) .
> Foldl.handles votes) Foldl.sum
We can combine all the filters and sums into one big query where they all run in parallel.
> getVotes :: Foldl.Fold ElectionResults [(Int, Int)]
> getVotes = sequenceA $ map (getPartyWard "Liberal Democrat") wardNames
I downloaded the geographical data from here but I imagine there are many other sources.
First we read all the shapefiles in the directory we downloaded.
> root :: FilePath
> root = "data/KingstonuponThames"
> main :: IO ()
> main = do
> ns <- runSafeT $ Foldl.purely fold getVotes (readTable "data/Election Results 2018 v1.csv")
> let ps :: [Double]
> ps = map (\(x, y) -> 100.0 * fromIntegral y / fromIntegral x) ns
> let percentLD = zip (map unpack wardNames) ps
>
> ds <- getDirectoryContents root
> let es = filter (\x -> takeExtension x == ".shp") ds
> let fs = case xs of
> Nothing -> error "Ward prefix does not match"
> Just ys -> ys
> where
> xs = sequence $
> map (fmap dropExtension) $
> map (stripPrefix "KingstonuponThames_") es
Sadly, when we get the ward names from the file names they don’t match the ward names in the electoral data sheet. Fortunately, fuzzy matching resolves this problem.
> rs <- loadElectionResults
> let wardSet :: Fuzzy.FuzzySet
> wardSet = Fuzzy.fromList . nub . Foldable.toList . fmap (^. ward) $ rs
>
> let gs = case sequence $ map (Fuzzy.getOne wardSet) $ map pack fs of
> Nothing -> error "Ward name and ward filename do not match"
> Just xs -> xs
Rather than making the colour of each ward vary continuously depending on the percentage share of the vote, let’s create 4 buckets to make it easier to see at a glance how the election results played out.
> let bucket :: Double -> String
> bucket x | x < 40 = "whitesmoke"
> | x < 50 = "yellow1"
> | x < 60 = "yellow2"
> | otherwise = "yellow3"
>
> let cs :: [String]
> cs = case sequence $ map (\w -> lookup w percentLD) (map unpack gs) of
> Nothing -> error "Wrong name in table"
> Just xs -> map bucket xs
Now we move into the world of R, courtesy of inline-r. First we load up the required libraries. Judging by urls like this, it is not always easy to install the R binding to GDAL (Geospatial Data Abstraction Library). But with nix, we don’t even have to use install.packages(...).
> R.runRegion $ do
> [r| library(rgdal) |]
> [r| library(ggplot2) |]
> [r| library(dplyr) |]
Next we create an empty map (I have no idea what this would look like if it were plotted).
> map0 <- [r| ggplot() |]
And now we can fold over the shape data for each ward and its colour to build the basic map we want.
> mapN <- foldM
> (\m (f, c) -> do
> let wward = root </> f
> shapefileN <- [r| readOGR(wward_hs) |]
> shapefileN_df <- [r| fortify(shapefileN_hs) |]
> withColours <- [r| mutate(shapefileN_df_hs, colour=rep(c_hs, nrow(shapefileN_df_hs))) |]
> [r| m_hs +
> geom_polygon(data = withColours_hs,
> aes(x = long, y = lat, group = group, fill=colour),
> color = 'gray', size = .2) |]) map0 (zip es cs)
Now we can annotate the map with all the details needed to make it intelligible.
> map_projected <- [r| mapN_hs + coord_map() +
> ggtitle("Kingston on Thames 2018 Local Election\nLib Dem Percentage Vote by Ward") +
> theme(legend.position="bottom", plot.title = element_text(lineheight=.8, face="bold", hjust=0.5)) +
> scale_fill_manual(values=c("whitesmoke", "yellow1", "yellow2", "yellow3"),
> name="Vote (%)",
> labels=c("0%-40% Lib Dem", "40%-50% Lib Dem", "50%-60% Lib Dem", "60%-100% Lib Dem")) |]
And finally save it in a file.
> [r| jpeg(filename="diagrams/kingston.jpg") |]
> [r| print(map_projected_hs) |]
> [r| dev.off() |]
> return ()
How I Really Use nix
I haven’t found a way of interspersing text with nix code so here are some comments.
-
I use neither all the Haskell packages listed nor all the R packages. I tend to put a shell.nix file in the directory where I am working and then
nix-shell shell.nix -I nixpkgs=~/nixpkgsas at the moment I seem to need packages which were recently added to nixpkgs. -
I couldn’t get the tests for
inline-rto run on MACos so I addeddontCheck. -
Some Haskell packages in nixpkgs have their bounds set too strictly so I have used
doJailbreak.
{ pkgs ? import {}, compiler ? "ghc822", doBenchmark ? false }:
let
f = { mkDerivation, haskell, base, foldl, Frames, fuzzyset
, inline-r, integration, lens
, pandoc-types, plots
, diagrams-rasterific
, diagrams
, diagrams-svg
, diagrams-contrib
, R, random, stdenv
, template-haskell, temporary }:
mkDerivation {
pname = "mrp";
version = "1.0.0";
src = ./.;
isLibrary = false;
isExecutable = true;
executableHaskellDepends = [
base
diagrams
diagrams-rasterific
diagrams-svg
diagrams-contrib
(haskell.lib.dontCheck inline-r)
foldl
Frames
fuzzyset
integration
lens
pandoc-types
plots
random
template-haskell
temporary
];
executableSystemDepends = [
R
pkgs.rPackages.dplyr
pkgs.rPackages.ggmap
pkgs.rPackages.ggplot2
pkgs.rPackages.knitr
pkgs.rPackages.maptools
pkgs.rPackages.reshape2
pkgs.rPackages.rgeos
pkgs.rPackages.rgdal
pkgs.rPackages.rstan
pkgs.rPackages.zoo];
license = stdenv.lib.licenses.bsd3;
};
haskellPackages = if compiler == "default"
then pkgs.haskellPackages
else pkgs.haskell.packages.${compiler};
myHaskellPackages = haskellPackages.override {
overrides = self: super: with pkgs.haskell.lib; {
diagrams-rasterific = doJailbreak super.diagrams-rasterific;
diagrams-svg = doJailbreak super.diagrams-svg;
diagrams-contrib = doJailbreak super.diagrams-contrib;
diagrams = doJailbreak super.diagrams;
inline-r = dontCheck super.inline-r;
pipes-group = doJailbreak super.pipes-group;
};
};
variant = if doBenchmark then pkgs.haskell.lib.doBenchmark else pkgs.lib.id;
drv = variant (myHaskellPackages.callPackage f {});
in
if pkgs.lib.inNixShell then drv.env else drv
Maths, Stats & Functional Programming 
Looks great! Mentioned in my tweet: https://twitter.com/giuliohome_2017/status/1110099413995872256